Method and system for near-eye three dimensional display

ABSTRACT

A 3D near eye display device is provided, the display device comprising a display screen for displaying more than one 2D images, at least one focusing element for collimating the images to sub-images, a spatial multiplexing unit capable of remapping the sub-images to different depths while forcing their centers to align to form remapped sub-images, and an eye piece. The device and methods allow for a high quality, compact 3D display system that can be wearable and overcomes the vergence-accomodation conflict that leads to visual discomfort and fatigue caused by traditional 3D near eye display devices.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application claims the benefit of U.S. Provisional Application No.62/363,886, filed Jul. 19, 2016, the disclosure of which is herebyincorporated herein by reference in its entirety.

FIELD OF THE DISCLOSURE

The subject disclosure relates to a method and system for near-eye threedimensional display.

BACKGROUND

Near-eye three-dimensional (3D) displays have seen rapid growth and heldgreat promise in a variety of applications, such as gaming, filmviewing, and professional scene simulations. Currently, most near-eyethree-dimensional displays are based on computer stereoscopy, whichpresents two images with parallax in front of the viewer's eyes.Stimulated by binocular disparity cues, the viewer's brain then createsan impression of the three-dimensional structure of the portrayed scene.However, these stereoscopic displays suffer from a major drawback of avergence-accommodation conflict, which reduces the viewer's ability tofuse the binocular stimuli while causing discomfort and fatigue. Thevergence-accomodation conflict can be attributed to the images beingdisplayed on one surface and the focus cues specifying the depth of thedisplay screen (i.e., accommodation distance) rather than the depths ofthe depicted scenes (i.e., vergence distance). This is opposite to theviewer's perception in the real world where these two distances arealways the same. To alleviate this problem, one must present correctfocus cues that are consistent with binocular stereopsis.

Currently, only a few approaches can attempt to provide correct ornearly correct focus cues for the depicted scene, such as light fieldnear-eye displays and multiplane near-eye displays. The light fielddisplay employs a lenslet array to project multi-view imagessimultaneously onto the viewer's retina, thereby yielding a continuousthree-dimensional sensation. Despite a compact form factor, the spatialresolution is low (˜100×100 pixels), restricted by the number of pixelsthat can fit into the imaging area of a lenslet.

By contrast, the multiplane display projects two-dimensional images ontoa variety of depth planes through either temporal multiplexing orspatial multiplexing. By synchronizing a fast display with a deformablemirror or a focal sweeping lens, the temporal-multiplexing-based methodsproject depth images in sequence. However, to render continuous motion,the device must display all depth images within the flicker fusion time( 1/60 s), thus introducing a severe trade-off between the image dynamicrange and the number of depth planes. For example, given 23 k patternrefresh rate at digital micromirror device and six displayed imageplanes, the dynamic range of each image for this process can beapproximately only 6 bits (64 grey levels). Alternatively, thespatial-multiplexing-based methods deploy multiple screens at variousdistances from the viewer, followed by optically combining their imagesusing a beam splitter. Because of the usage of multiple screens, suchdevices are normally bulky, making them unsuitable for wearable devices.

BRIEF DESCRIPTION OF THE DRAWINGS

The features, objects and advantages other than those set forth abovewill become more readily apparent when consideration is given to thedetailed description below. Such detailed description makes reference tothe following drawings, wherein:

FIG. 1 is a block diagram illustrating an example embodiment of anoperating principle of an optical mapping near-eye three dimensionaldisplay utilizing a spatial multiplexing unit in accordance with variousaspects described herein.

FIG. 2 is a block diagram illustrating an example embodiment of anoptical mapping near-eye three dimensional display utilizing a spatiallight modulator and an organic light emitting diode in accordance withvarious aspects described herein.

FIG. 3 is an illustration of visualization of intermediate depth planeimages including: (a) Optical mapping of four letters “U,” “I,” “U,” “C”from four sub-panels to the central field of view; and (b-e)Intermediate depth images captured at 0D, 1D, 2D, and 3D, respectively.

FIG. 4 is a graphical representation of an evaluation of an OMNI displayincluding: (a) Modulation contrast at a given spatial frequency (5lp/mm) versus depth plane spacing (A); and (b) Modulation contrast at agiven spatial frequency (5 lp/mm) versus the accommodation distance (z),where the modulation contrasts were calculated using a standard slantededge method.

FIG. 5 is an OMNI display of a complex three-dimensional sceneincluding: (a) Ground-truth all-in-focus image; (b) Ground-truth depthmap; (c) Representative depth image captured at 0D; and (d)Representative depth image captured at 3D.

FIG. 6A is a block diagram illustrating an example embodiment of anoptical mapping near-eye three dimensional display utilizing aphase-only spatial light modulator for virtual reality in accordancewith various aspects described herein.

FIG. 6B is a block diagram illustrating an example embodiment of anoptical mapping near-eye three dimensional display utilizing aphase-only spatial light modulator for augmented reality in accordancewith various aspects described herein.

FIG. 7a is a block diagram illustrating an example embodiment of anoptical mapping near-eye three dimensional display utilizing a volumeholography grating for virtual reality in accordance with variousaspects described herein.

FIG. 7b is a block diagram illustrating an example embodiment of anoptical mapping near-eye three dimensional display utilizing a volumeholography grating for augmented reality in accordance with variousaspects described herein.

FIGS. 8a, b is a block diagram illustrating an example embodiment ofimaging through a distorted phase grating in accordance with variousaspects described herein.

FIG. 9a is a block diagram illustrating an example embodiment of anoptical mapping near-eye three dimensional display utilizing a distortedphase grating for virtual reality in accordance with various aspectsdescribed herein.

FIG. 9b is a block diagram illustrating an example embodiment of anoptical mapping near-eye three dimensional display utilizing a distortedphase grating for augmented reality in accordance with various aspectsdescribed herein.

FIG. 10a is a block diagram illustrating an example embodiment of anoptical mapping near-eye three dimensional display utilizing a foldedflexible display screen for virtual reality in accordance with variousaspects described herein.

FIG. 10b is a block diagram illustrating an example embodiment of anoptical mapping near-eye three dimensional display utilizing a foldedflexible display screen for augmented reality in accordance with variousaspects described herein.

While the present disclosure is susceptible to various modifications andalternative forms, exemplary embodiments thereof are shown by way ofexample in the drawings and are herein described in detail. It should beunderstood, however, that the description of exemplary embodiments isnot intended to limit the disclosure to the particular forms disclosed,but on the contrary, the intention is to cover all modifications,equivalents and alternatives falling within the spirit and scope of thedisclosure as defined by the embodiments above and the claims below.Reference should therefore be made to the embodiments above and claimsbelow for interpreting the scope of the disclosure.

DETAILED DESCRIPTION

The devices and methods now will be described more fully hereinafterwith reference to the accompanying drawings, in which some, but not allembodiments of the disclosure are shown. Indeed, the exemplaryembodiments may be embodied in many different forms and should not beconstrued as limited to the embodiments set forth herein; rather, theseembodiments are provided so that this disclosure will satisfy applicablelegal requirements.

Likewise, many modifications and other embodiments of the devices andmethods described herein will come to mind to one of skill in the art towhich the disclosure pertains having the benefit of the teachingspresented in the foregoing descriptions and the associated drawings.Therefore, it is to be understood that the disclosure is not to belimited to the specific embodiments disclosed and that modifications andother embodiments are intended to be included within the scope of theappended claims. Although specific terms are employed herein, they areused in a generic and descriptive sense only and not for purposes oflimitation.

Unless defined otherwise, all technical and scientific terms used hereinhave the same meaning as commonly understood by one of skill in the artto which the disclosure pertains. Although any methods and materialssimilar to or equivalent to those described herein can be used in thepractice or testing of the present disclosure, the preferred methods andmaterials are described herein.

One or more of the exemplary embodiments provide a 3D near-eye displaydevice and methods that can be used in virtual-reality and/oraugmented-reality wearable devices, such as smart glasses or helmets. Byremapping different portions of a 2D image to different depths in 3Dspace, the proposed method solves a vergence-accomodation conflict,which can be the main cause for visual discomfort and fatigue. Inaddition, the resultant system is compact and light-weight, facilitatingits integration with wearable devices. In one or more embodiments,near-eye 3D displays can be applied to scientific and/or medicalvisualization. In one or more embodiments, to alleviate the problem ofthe vergence-accomodation conflict, a device can present correct focuscues that are consistent with binocular disparity cues.

In one or more embodiments, an optical mapping near-eye (OMNI)three-dimensional display method, such as for wearable devices, isprovided. For example, by dividing a display screen into differentsub-panels and optically mapping them to various depths, a multiplanevolumetric image can be created or otherwise generated with correctfocus cues for depth perception. The resultant system can drive theeye's accommodation to the distance that is consistent with binocularstereopsis, thereby reducing or eliminating the vergence-accommodationconflict, which can be the primary cause for eye fatigue and discomfort.Compared with conventional methods, the OMNI display offers prominentadvantages in adaptability, image dynamic range, and refresh rate.

In one or more embodiments, a spatial-multiplexing-based multi-planedisplay device is described. However, rather than using multipleelectronic screens, one or more of the exemplary embodiments can employonly one screen, maintaining a compact form factor similar totemporal-multiplexing-based methods. In one or more embodiments, thesystem maps different portions of a display screen to different depthswhile forcing their centers aligned. The multiplane images can then bereimaged onto a viewer's retina through an eyepiece and a focusingelement (e.g., crystalline lens). Capable of providing the correct focuscues, this approach can solve the aforementioned vergence-accomodationconflict seen in conventional computer stereoscopic displays.Furthermore, unlike the temporal-multiplexing-based methods, the imagedynamic range and the number of displayed planes can be decoupled in oneor more of the exemplary embodiments. Therefore, the resultant systemcan form 3D images with a high dynamic range limited by only theelectronic screen itself.

In one or more embodiments, a 3D near eye display device includes adisplay screen that displays a plurality of two dimensional (2D) images,a focusing element that collimates the plurality of 2D images, a spatialmultiplexing unit (SMU) that remaps the plurality of 2D images todifferent depths while forcing centers of the plurality of 2D images toalign, and an eye piece.

In one or more embodiments, a method can include displaying, by adisplay screen of a wearable device, a plurality of 2D images. Themethod can include collimating, by a focusing element of the wearabledevice, the plurality of 2D images. The method can include modifying, bya SMU of the wearable device, a phase of incident light by addingquadratic and linear phase terms to an incident wave front of the 2Dimages resulting in multiplane images. The method can include reimaging,by an eye piece of the wearable device, the multiplane images onto aviewer's retina.

In one or more embodiments, a 3D near eye display device includes adisplay screen comprising first and second panels that display 2Dimages. The 3D near eye display device includes a beam splitter havingfirst and second surfaces that align with the first and second panels,respectively, where the beam splitter combines light emanating from thefirst and second panels of the display screen. The 3D near eye displaydevice includes an actuator that laterally slides the beam splitter in adirection along the first panel of the display screen to adjust a gapbetween the second panel and the second surface of the beam splitterresulting in an optical path difference. The 3D near eye display deviceincludes a focusing element that projects the 2D images to differentdepth planes and includes an eye piece.

In one aspect, one or more of the exemplary embodiments provide a 3Dnear-eye display device comprising a display screen, at least onefocusing element, a SMU, and an eye piece. In one embodiment, thedisplay screen is selected from light emitting diode (LED), liquidcrystal display (LCD), organic light emitting diode (OLED) and flexibleOLED. In another embodiment, the display screen may not be provided bythe device, but provided by a secondary source (e.g. smart phone,tablet). For example, the 3D near eye device may be designed so as tolack a display screen, but have the ability to attach a portable devicesuch as a smart phone or a tablet to the device, which acts as thedisplay screen to display the 2D images. In this embodiment, the devicewill provide a means of securely attached a secondary screen such as acell phone to the device. The display screen can be any screen capableof delivering an image, for example, it could be a mirror reflecting animage or a rasterized image where a beam draws an image like the olderCRT TVs. In addition, the display screen may also be multiple abuttingdisplays.

In another embodiment, the device is wearable, e.g. in the form ofgoggles, glasses, or a helmet. In another embodiment, the 2D imagescomprise at least two 2D images, with the images displayed at a varietyof depths. The number of images does not need to be limited. In theory,the more images the better. However, there is a trade-off between eachimages pixel resolution and the total number of images. In oneembodiment, the display screen displays between 2 to 6 images. In afurther embodiment, the device further comprises computer hardwarerunning software to execute algorithms to assist in reimaging andmodifying the incident light by the SMU and for remapping the images. Inone embodiment, a depth-weighted blending algorithm can be used todetermine the content of sub-images at the display screen.

In one embodiment, the SMU can be a special light modulator (SLM), andcan include, but is not limited to, liquid-crystal-on-silicon (LCOSD)special light modulator, a volume holography grating (VHG) and adistorted phase grating. The SMU can be used to map sub-images todifferent depths. Mapping sub-images to different depths while forcingtheir centers aligned faces two challenges: lateral remapping anddefocus compensation. To achieve these two goals, the SMU can functionas multifocal off-axis Fresnel lenses, adding both the linear andquadratic phase terms to the incident wavefront.

In some embodiments, the device comprises more than one focusingelement. The number of focusing elements used can vary depending on, forexample, what type of SMU is utilized. When the SMU is LCOS SLM (e.g.,FIG. 6A) or VHG (e.g., FIG. 7a ), the device may comprise a secondand/or a third focusing element. In some embodiments, a first focusingelement can be positioned between the display screen having more thanone image of varying depth and the SMU, and this focusing element canfunction to collimate the images from the display screen into the SMU,with their centers aligning. A second focusing element may be employedto take the remapped images from the SMU and help focus them intosub-images that are viewable by the pupil of a viewer either by aneyepiece (for virtual reality), or an eyepiece attached to a viewcombiner (for augmented reality). In another embodiment, a thirdfocusing element may be used to help focus the image for better viewingthrough the eyepiece. Alternatively if the SMU is a distorted phasegrating (e.g., FIG. 9a ) or makes use of a folded flexible displayscreen and a beam splitter (e.g., FIG. 10a ), only one focusing elementmay be used. For a distorted phase grating, the focusing element can bepositioned a distance from the display screen, with the distorted phasegrating located immediately adjacent to the focusing element. Thefocusing element and distorted phase grating projects intermediateimages that can be viewed by the eyepiece, or by the eyepiece attachedto a view combiner.

For a flexible display screen configuration, the display screen can bebent at about a 90° angle with at least one image on each surface (FIG.10a ), and a beam splitter located close to the folded display screen. Afocusing element can be positioned between the beam splitter and theeyepiece (with or without a waveguide). The view combiner can include,but is not limited to, a beam splitter, a prism, or a waveguide.“Focusing element” can include any materials or arrangement ofmaterials, which can be used to focus an optical image. Examples offocusing elements include, but are not limited to, lenses, crystallinelenses, converging and diverging prisms, a magnetic field used forfocusing electron beams, a focusing mirror alone or in combination withfocusing lenses, and the like. In one embodiment, the focusing elementis a lens. In another embodiment, the lens is a crystalline lens.Crystalline lenses are the most common focusing element, however, itshould be noted that the exemplary embodiments are not limited tocrystalline lenses and any element or means that can focus an image canbe used in the present disclosure.

The operating principle for an exemplary OMNI 3D display 100 isillustrated in FIG. 1. A high-resolution two-dimensional image can bedisplayed at an electronic screen 110. The image can consist of severalsub-panels, each targeting to be displayed at a designated depth. In oneembodiment, an optical relay (e.g., a 4f optical relay) can be utilizedwhich has a spatial multiplexing unit (SMU) 120 located at the Fourierplane. The SMU 120 can function as a multifocal off-axis Fresnel lens,adding both quadratic and linear phase terms to the incident wavefront.The quadratic phase terms can axially shift sub-panel images to thedesignated depths, while the linear phase terms can laterally shift thecenters of sub-panel images to the optical axis. As a result, thesub-panel images can be mapped to different axial locations andlaterally aligned at the output end. Finally, the light emanated fromthese intermediate depth images can be collected by an eyepiece 130 andenters the eye pupil. Depending on their relative axial positions, theviewer perceives these multi-depth images at a distance from a nearplane to infinity. Unlike previous spatial multiplexing approaches, OMNI3D display 100 can utilize a single display screen at the input, therebymaintaining a compact form factor.

In one or more embodiments, the electronic screen 110 can be a liquidcrystal display, light emitting diode, or organic light emitting diode.In one or more embodiments, the light emanated from the electronicscreen 110 can be collimated by a lens 115. In one or more embodiments,the SMU 120 can be a transmissive or reflective SMU. In one or moreembodiments, the SMU 120 can be placed at the exit pupil of the lens115, modifying the phase of incident light. In one or more embodiments,the transmitted or reflected light can be collected by a second lens125, forming multiple images at different depths. In one or moreembodiments, the resultant multi-plane images when reimaged have a backfocal plane that co-locates with the viewer's eye pupil. In one or moreembodiments, to create a 3D scene with continuous depth perception, adepth-weighted blending algorithm can be employed to determine thecontent of sub-images at the display screen.

Referring to FIG. 2, the OMNI 3D display 200 can include a reflectiveSMU that employs a liquid-crystal-on-silicon spatial light modulator220. In one embodiment, light emanated from a monochromatic organiclight emitting diode screen 210 (e.g., MDP02BCYM, 2000×2000 pixels,Micro oled) can be filtered in color (e.g., central wavelength, 550 nm;bandwidth, 10 nm) and/or polarization (e.g., p light), such as throughuse of color filter 212 and linear polarizer 214, respectively. Thefiltered light can pass through a beam splitter 230 (e.g., a 50:50 beamsplitter). The light can then be collimated by an infinity-correctedmicroscope objective (e.g., 2× M Plan APO, Edmund Optics). The LCOS-SLM220 can be positioned at the exit pupil of the objective lens 225 tomodulate a phase of the incident light. The reflected light can becollected by the same objective lens 225, reflected at the beam splitter230, and can form intermediate images at a variety of depths in front ofan eyepiece 250 (e.g., focal length, 25 mm; LB1761-A, Thorlabs).

To map the sub-panel image to the designated location, a phase patternis displayed on the LCOS-SLM in the form:

${{\phi_{i}\left( {x,y} \right)} = {\frac{\pi \left( {x^{2} + y^{2}} \right)}{\lambda \; f_{i}} + {\frac{2\pi}{\lambda}\left\lbrack {{{\sin \left( \frac{l_{x_{i}}}{f_{o}} \right)}x} + {{\sin \left( \frac{l_{y_{i}}}{f_{o}} \right)}y}} \right\rbrack}}},$

where λ is the light wavelength, f_(i) is the effective focal length ofthe LCOS-SLM, f_(o) is the focal length of the objective lens in FIG. 2,l_(x) _(i) and l_(y) _(i) are center coordinates of sub-panel image i atthe OLED. Because each sub-panel image requires a different set off_(i), l_(x) _(i) , and l_(y) _(i) , the ideal phase pattern thatenables simultaneous mapping of all sub-panel images is Σ_(i)φ_(i).However, in practice, since the displayed phase must be wrapped within2π and discretized into 8-bit levels, the simple additive phase patternis inapplicable.

To generate a phase pattern that functions similarly to Σ_(i)φ_(i), anoptimization algorithm, Weighted Gerchberg-Saxton (WGS), can be applied.WGS starts with an initial phase estimate φ_(est)(x, y), followed byiteratively updating this estimate to maximize a merit function T, whichis defined as:

$T = {\sum\limits_{i = 1}^{A}\; {\left\{ {\frac{1}{B}{\sum_{x}{\sum_{y}e^{j{\lbrack{{\phi_{est}{({x,y})}} - {\phi_{i}{({x,y})}}}\rbrack}}}}} \right\}.}}$

Here x, y are the discretized Cartesian coordinates, A is the totalnumber of sub-panel images, B is the total number of the LCOS-SLM'spixels, and j is the imaginary number. The optimization processmaximizes the overall likelihood between φ_(est) and φ_(i) for allsub-panel images.

In one or more embodiments to create a three-dimensional scene withcontinuous depth perception, a linear depth-weighted blending algorithmcan be applied to create the contents of sub-panel images. The imageintensity at each depth plane can be rendered proportional to thedioptric distance of the point from that plane to the viewer along aline of sight. The sum of the image intensities can be maintained as aconstant at all depth planes.

Compared with existing near-eye three-dimensional displays, the OMNIdisplay 200 offers advantages in adaptability, image dynamic range,and/or refresh rate. In one embodiment, because the sub-panel imagesoccupy the same display screen at the input end, the product of a depthplane's lateral resolution (L×M pixels) and the number of depth planes(N) should not be greater than the total number of pixels (P) at thedisplay screen, i.e., L×M×N≦P. Taking this constraint intoconsideration, the system can be configured working in two modes whichselectively bias the lateral resolution and the depth plane spacing,respectively, by alternating the phase patterns on the LCOS-SLM. For thegiven high-resolution OLED (F=4 megapixels) and a depth range of 0-3D(diopter), two typical display settings are summarized in

TABLE 1 Table 1. System parameters of an OMNI display. Lateralresolution Depth plane spacing (pixels) (diopter) High lateral 1000 ×1000 1.0 resolution mode Dense depth 500 × 500 0.2 sampling mode

The scalability of display parameters thus grants more freedom to adaptthe OMNI display 200 to the depicted scene framewise. Furthermore,unlike the temporal-multiplexing-based multiplane display, herein theimage refresh rate and dynamic range are decoupled from the number ofdepth planes and thereby limited by only the display screen itself.Using the given OLED, a high dynamic range (12 bits) three-dimensionalvideo can be displayed in real time (30 Hz).

Referring to FIG. 3, to visualize the intermediate depth images in theOMNI display 200, a mapping experiment was performed. At the input end,four letters “U”, “I”, “U”, “C” are displayed in the four sub-panels ofthe OLED (FIG. 3(a)) and the OMNI display 200 was set working in thehigh lateral resolution mode (

Table 1).

A camera was placed at the focus of the eyepiece and translated towardsthe eyepiece, to capture images at four nominal depth planes (0D, 1D,2D, and 3D). The remapped letter images at these four depths are shownin FIGS. 3(b)-(e), respectively. The letters appear sharp at theirdesignated depths while blurred elsewhere.

The depth plane spacing and accommodation distance were then varied toevaluate their effects on the image contrast. In this mappingexperiment, a camera was positioned in front of the eyepiece to mimic aneye that focused at 1.5D. Two sub-panel images (a slanted edge) weredisplayed at the OLED and they were projected to (1.5+Δz/2)D and(1.5−Δz/2)D, respectively. The depth plane spacing was varied, Δz,through changing the effective focal length f_(i) in the phase patternat the LCOS-SLM (Equation shown above). Accordingly, depth-fused imageswere acquired at the camera, and the image modulation contrast wascalculated. The dependence of modulation contrast on the depth planespacing Δz is shown in FIG. 4a . The modulation contrast degrades as thedepth plane spacing increases. However, for a given number of depthplanes, decreasing the depth plane spacing will unfavorably reduce thetotal depth range. Therefore, one should balance the image quality for adesired depth range.

In another mapping experiment, the focal depth was varied, z, of thecamera to mimic the accommodation distance change of the eye. At theinput end, two identical images were displayed at the 1D and 2D depthplane. Because the light intensities along a line of sight at these twodepth planes are identical, the rendered depth is at the dioptricmidpoint, 1.5D. Images were captured at a variety of dioptricaccommodation distances and the corresponding image modulation contrastswere derived. The result (FIG. 4b ) shows that the modulation contrastreaches the maximum at z=1.5D and degrades smoothly around this depth.Since the human eye inherently focuses on the depth that provides thehighest modulation contrast, OMNI display 200 thereby provides a correctfocus cue that can drive the eye's accommodation to the desired depth.

The OMNI display 200 was also tested using a complex three-dimensionalscene. The ground-truth all-in-focus image and the corresponding depthmap are shown in FIGS. 5(a) and (b), respectively. The display contentswere generated at four nominal depth planes (0D, 1D, 2D, and 3D). Thefocal depth of the camera was varied to mimic the accommodation distancechange of the eye. The depth-fused images captured at a far plane (0D)and a near plane (3D) are shown in FIGS. 5(c) and (d), respectively,matching closely with the ground-truth depth map (FIG. 5(b)).

In the OMNI display 200, an LCOS-SLM was utilized as the SMU toaccomplish the optical mapping. However, the SMU can also be other phasemodulation devices, such as a volume holography grating or a distortedphase grating. Similar to the LCOS-SLM, both these phase modulators canact as a multifocal off-axis Fresnel lens, directing the sub-panelimages to the designated depths while forcing their centers aligned.However, unlike the LCOS-SLM, the volume holograph grating and distortedphase grating are passive devices. Passive phase modulators require nopower supplies, reducing the system volume as well as power consumption.However, because their phase patterns are stationary, passive phasemodulators may not scale the display parameters in the adaptive fashionas previously discussed.

The OMNI display 200 can reproduce colors. Using a white light OLED asthe input screen, a sub-panel image can be split into three channels,followed by covering them with a red, green, and blue color filter,respectively. Accordingly, at the LCOS-SLM, a phase pattern is displayedthat compensates for the wavelength difference, thereby mapping thesefiltered images to the same depth. Nevertheless, given a desired depthplane spacing, displaying colors will unfavorably reduce the lateralresolution by a factor of three.

The OMNI display 200 provides an optical mapping near-eyethree-dimensional display method with correct focus cues that areconsistent with the binocular vision, thus eliminating thevergence-accommodation conflict. Through mapping different sub-panelimages of a display screen to various axial depths, a high-resolutionthree-dimensional image is created over a wide depth range. The imagedynamic range and refresh rate may be limited by only the display screenitself, such as up to 12 bits and 30 Hz, respectively.

FIGS. 6a, 6b illustrate an implementation of a phase-only reflectiveLCOS spatial light modulator for virtual reality (FIG. 6a ) andaugmented reality displays (FIG. 6b ). In FIG. 6a , the light emanatedfrom the display screen 610 passes through the beam splitter 630 wherethe transmitted light is then collimated by a lens 625. The SLM 620locates at the Fourier plane of display screen 610, modifying the phaseof incident light. The reflected light is then collected by the samelens 625. Half of the light is reflected at the beam splitter 630,forming intermediate sub-images close to the lens' back focal plane.These sub-images are then collimated by an eyepiece 650 and reimagedonto the viewer's retina by the crystalline lens. The optical setup inFIG. 6b is similar to FIG. 6a which is similar to the optical setup inFIG. 2. However in the embodiment of FIG. 6b , after being collimated bythe eyepiece 650, the light is directed into a waveguide 660 andtransferred to the eye pupil. Since the waveguide 660 is transparent,the light emanated from the real-world objects can also enter the eyepupil, forming an image overlaid with the virtual objects. In one ormore embodiments, other view combiner devices can be used in place ofthe waveguide 660, such as a beam splitter or a prism.

To provide an example, the display of FIG. 6a is described using onlyoff-the-shelf components. The system projects four virtual images ontoplanes which are 0, 1, 2, and 3 diopters from the viewer. Based on thethin lens equation, the depth spacing, Au, between adjacent images atthe object side of eyepiece is:

$\begin{matrix}{{\Delta \; u} = {\frac{\Delta \; D}{\left( {{1/f_{e}} + D} \right)^{2}}.}} & (1)\end{matrix}$

where ΔD is depth spacing in diopters between adjacent virtual images, Dis depth in diopters, and f_(e) is the focal length of eyepiece. Toseparate these images along the depth axis, their spacing Δu must begreater than the lens' depth of focus, i.e.

$\begin{matrix}{{\Delta \; u} > {\frac{2\lambda}{\pi \; {NA}^{2}}.}} & (2)\end{matrix}$

Here NA is the numerical aperture of the lens. Combining Eq. 1 and Eq. 2yields

$\begin{matrix}{{NA} > {\sqrt{\frac{2\lambda}{{\pi\Delta}\; D}}{\frac{1}{f_{e}}.}}} & (3)\end{matrix}$

To mimic an off-axis Fresnel lens with focal length f_(SLM) the SLMdisplays a phase pattern which can be mathematically described by:

$\begin{matrix}{\phi = {\frac{\pi \; r^{2}}{\lambda \; f_{SLM}} + {\frac{2\pi}{\lambda}{\left( {{\sin \; \theta_{x}x} + {\sin \; \theta_{y}y}} \right).}}}} & (4)\end{matrix}$

At the display screen, sub-images are displayed. The size of eachsub-image is l₁×l₂ mm². To shift the centers of sub-images to theoptical axis, the SLM must divert the chief rays towards the directions:

$\begin{matrix}{{\theta_{x} \sim \frac{l_{1}}{2\; f_{l}}},{\theta_{y} \sim {\frac{l_{2}}{2\; f_{l}}.}}} & (5)\end{matrix}$

Given small angle approximation, combining Eq. 4 with Eq. 5 gives:

$\begin{matrix}{\phi = {\frac{\pi \; r^{2}}{\lambda \; f_{SLM}} + {\frac{\pi}{\lambda \; f_{l}}\left( {{l_{1}x} + {l_{2}y}} \right)}}} & (6)\end{matrix}$

The phase gradients along the x and y axes are

$\begin{matrix}{{\frac{\Delta\phi}{\Delta \; x} = {\frac{2\pi \; x}{\lambda \; f_{SLM}} + \frac{\pi \; l_{1}}{\lambda \; f_{l}}}}{\frac{\Delta\phi}{\Delta \; y} = {\frac{2\pi \; x}{\lambda \; f_{SLM}} + {\frac{\pi \; l_{2}}{\lambda \; f_{l}}.}}}} & (7)\end{matrix}$

In practice, the phase change across a pixel must be smaller than 0.5π(i.e., 2π phase is sampled by four SLM pixels):

$\begin{matrix}{{{\left( {\frac{2\pi \; x}{\lambda \; f_{SLM}} + \frac{\pi \; l_{1}}{\lambda \; f_{l}}} \right)P} < {0.5{\pi \left( {\frac{2\pi \; x}{\lambda \; f_{SLM}} + \frac{\pi \; l_{2}}{\lambda \; f_{l}}} \right)}P} < {0.5\pi}},} & (8)\end{matrix}$

where P is the dimension of the pixel. From Eq. 7, we can get,

$\begin{matrix}{{l_{1} < {f_{l}{\lambda/2}\; P}}{l_{2} < {f_{l}{\lambda/2}\; P}}{f_{SLM} > {\frac{4\; x_{\max}P}{\lambda}.}}} & (9)\end{matrix}$

Reimaged by the eyepiece, the final angular FOV is

FOV=√{square root over (l ₁ ² +l ₂ ²)}/f _(e)  (10)

System components are illustrated in Table 2, which were determinedbased on the design constraints described by Eq. 3, 9, and 10, and thecorrespondent first-order design parameters are summarized in Table 3:

TABLE 2 System components based on design constrains described by Eq. 3,9, and 10. Display MICROOLED, MDP02, 2600 × 2088 pixels, pixel pitch,screen 4.7 um. Sensor size, 12.2 mm × 9.8 mm. Lens Edmund Optics, 2X EOM Plan Apo Long Working Distance Infinity Corrected, f_(l) = 100 mm, NA= 0.055, WD = 34 mm, FOV = 12 mm (8.5 mm × 8.5 mm), aperture diameter,11 mm. SLM Holoeye, GAEA-VIS-036 Phase Only Spatial Light Modulator(420-650 nm), pixel pitch P = 3.74 μm, frame rate, 25 Hz, phase noise,0.06 π. Eyepiece Edmund Optics, f_(e) = 12 mm Mounted, RKE PrecisionEyepiece

TABLE 3 Correspondent first-order design parameters. Image resolution l₁= 4.25 mm, l₂ = 4.25 mm, 900 × 900 pixels. Angular FOV ~29 degrees Focallengths of 23.981, 36.404, 73.671, and infinity SLM (m) Quadratic phase2.2934 π, 1.5108 π, 0.7466 π, 0 (must be greater than term at the edge10x phase noise, 0.06 π) of pupil Linear phase 0.29 π/pixel (must besmaller than 0.5 π/pixel) gradient Perception depths 0, 1, 2, 3 dioptersfor the viewer

FIGS. 7a,b illustrate an implementation of the near-eye 3D display usinga volume holography grating 720 for virtual reality (FIG. 7a ) andaugmented reality displays (FIG. 7b ). In one embodiment, the electronicscreen 710 can be divided into a number of panels (e.g., four), eachdisplaying a sub-image. This implementation utilizes a volume hologram'swavefront selection properties to simultaneously project multiplesub-images to different depths. In FIG. 7a , the volume hologram isplaced at the aperture stop of a lens 715, acting as a Bragg filter andallowing photons with only specific propagation angles and wavelengthsto pass through. To enable simultaneous projection of multiple planes,the volume hologram can be produced in a multiplexed manner—superimposedby holographic gratings 720 with different frequency patterns. Eachmultiplexed grating can be Bragg matched to a different depth anddiffracts the light to a different central angle. After passing throughthe volume hologram, the diffracted light is collected by a second lens725, forming images at different depths close to the back focal plane ofthe eyepiece 750. Finally, these images are collimated by the eyepiece750 and reimaged onto the viewer's retina by the crystalline lens.

The optical setup in FIG. 7b is similar to FIG. 7a . However, afterbeing collimated by the eyepiece 750, the light is directed into awaveguide 760 and transferred to the eye pupil. Since the waveguide 760is transparent, the light emanated from the real-world objects can alsoenter the eye pupil, forming an image overlaid with the virtual objects.In one or more embodiments, other view combiner devices, such as a beamsplitter or a prism, can replace the waveguide 760.

Referring to FIGS. 8a,b , a distorted phase grating can introducedifferent levels of defocus in the wavefront and diffract them intodifferent orders. Therefore, when a distorted grating is placed close toa lens, it effectively modifies the focal length of the lens in non-zerodiffraction orders, playing the role of a defocus compensator.Additionally, the diffraction angles enable depth remapping. The effectof a distorted phase grating on an imaging system is illustrated inFIGS. 8a,b . The combination of a distorted grating and a lens images asingle object onto different depth planes in each different diffractionorder is shown in FIG. 8a . If multiple objects are located at the sameplane but different lateral positions, one or more embodiments cansimultaneously image them onto multiple depths while forcing theircenters aligned. In FIG. 8b , the three depth images correspond to theobjects A, B, C associated with the −1, 0, and +1 diffraction orders,respectively.

FIGS. 9a, b illustrate an embodiment of a near-eye 3D display using adistorted phase grating 920 for virtual reality (FIG. 9a ) and augmentedreality displays (FIG. 9b ). The electronic screen 910 can be dividedinto a number of panels (e.g., four), each displaying a sub-image. Adistorted phase grating 920 is closely placed next to a lens 915,alternating the effective focal length of the lens for differentdiffraction orders. After passing through the distorted phase grating920 and the lens 915, the diffracted light is focused to differentdepths which are close to the back focal plane of the eyepiece 950.Finally, these images are collimated by the eyepiece 950 and reimagedonto the viewer's retina by the crystalline lens. The optical setup inFIG. 9b is similar to FIG. 9a . However, after being collimated by theeyepiece 950, the light is directed into a waveguide 960 and transferredto the eye pupil. Since the waveguide 960 is transparent, the lightemanated from the real-world objects can also enter the eye pupil,forming an image overlaid with the virtual objects. In one or moreembodiments, the waveguide 960 can be replaced by other view combinerdevices, such as a beam splitter or a prism.

FIGS. 10a, b illustrate an embodiment of a near-eye 3D display using aflexible electronic screen 1010 for virtual reality (FIG. 10a ) andaugmented reality displays (FIG. 10b ). In One or more embodiments, theflexible electronic screen 1010 (e.g., a flexible OLED) can be foldedinto a number of panels (e.g., panels 1 and 2), each aligning with asurface of a beam splitter 1030. Each panel displays an image. The lightemitting from these two panels is combined by the beam splitter 1030. Anoptical path difference is introduced between these two imaging arms bylaterally sliding the beam splitter 1030 along panel 1 of the displayscreen 1010, creating a gap between panel 2 and the beam splitter. Thedisplayed panel images are then projected to different depth planes by alens 1025 and finally onto the viewer's retina by an eyepiece 1050 andthe crystalline lens. In another embodiment, two separate electronicscreens can be utilized that may or may not be flexible. The opticalsetup in FIG. 10b is similar to FIG. 10a . However, after beingcollimated by the eyepiece 1050, the light is directed into a waveguide1060 and transferred to the eye pupil. Since the waveguide 1060 istransparent, the light emanated from the real-world objects can alsoenter the eye pupil, forming an image overlaid with the virtual objects.In FIG. 6b , the waveguide 1060 can be replaced by other view combinerdevices, such as a beam splitter or a prism.

In one or more embodiments, a 3D near eye display device can include acomputer running software to execute algorithms to modulate wavefrontsand assist in remapping images. In one or more embodiments, a 3D neareye display device can include a SMU modulating the phase and/oramplitude of incident light. In one or more embodiments, a 3D near eyedisplay device can include a second focusing element capable of mappingdifferent portions of the display screen to different depths whileforcing their centers to align. In one or more embodiments, a 3D neareye display device can include a third focusing element located betweena second focusing element and an eyepiece, where the third focusingelement focuses a remapped image into the eyepiece. In one or moreembodiments, a 3D near eye display device can be wearable. In one ormore embodiments, a 3D near eye display device can be in the shape of ahelmet, goggles or glasses. In one or more embodiments, a 3D near eyedisplay device can be used for virtual reality or augmented reality. Inone or more embodiments, a 3D near eye display device can include adisplay screen displaying at least 2 sub-images, each sub-imagedisplayed at different depths. In one or more embodiments, a 3D near eyedisplay device can include a SMU modifying the phase of incident lightand adding quadratic and linear phase terms to the incident wave front.In one or more embodiments, a 3D near eye display device can displaydifferent portions of a display screen to different depths while forcingtheir centers aligned. In one or more embodiments, a 3D near eye displaydevice can form 3D images with a high dynamic range limited by only thedisplay screen itself. In one or more embodiments, a 3D near eye displaydevice can be configured where the display screen is provided by asecondary device. In one or more embodiments, the secondary device is acell phone or a tablet.

In one or more embodiments, a 3D near eye display device can include aflexible display screen for displaying a 2D image, a beam splitter, aSMU, a focusing element, and an eye piece. In one or more embodiments,the beam splitter can be located in close proximity to a flexibledisplay screen folded into a 90 degree angle and displaying at least oneimage on each side the folded screen, where the focusing element islocated between the eyepiece and the flexible display screen. In one ormore embodiments, the flexible display screen can be a flexible OLED.

In one or more embodiments, a near-eye 3D augmented reality display caninclude a display screen capable of displaying more than one 2D image ofvarying depth, a first focusing element capable of collimating theimages from the display screen to form a plurality of sub-images, a SMUcapable of remapping the plurality of sub-images to different depthsforcing their centers to align to form remapped sub-images, a secondfocusing element capable of focusing the remapped sub-images, an eyepiece, and a view combiner. In one or more embodiments, the viewcombiner optically combines the computer simulated 3D image withreal-world objects.

In one or more embodiments, a method of 3D near-eye display can includeremapping different portions of a display screen comprising multipleimages to different depths to create a plurality of sub-images, wherethe centers of the plurality of sub-images are aligned, and reimagingthe plurality of sub-images into an eyepiece.

One or more of the exemplary embodiments described herein provide anOMNI three-dimensional display method which provides correct focus cuesfor depth perception. One or more of the exemplary methods map differentportions of a display screen to various depths while forcing theircenters aligned. These intermediate depth images can then be reimaged byan eyepiece and projected onto the viewer's retina.

One or more embodiments can include a machine in the form of a computersystem within which a set of instructions, when executed, may cause themachine to perform any one or more of the methods described above. Insome embodiments, the machine may be connected (e.g., using a network)to other machines.

The computer system may include a processor (or controller) (e.g., acentral processing unit (CPU)), a graphics processing unit (GPU, orboth), a main memory and a static memory, which communicate with eachother via a bus. The computer system may include an input device (e.g.,a keyboard), a cursor control device (e.g., a mouse), a disk drive unit,a signal generation device (e.g., a speaker or remote control) and anetwork interface device. The disk drive unit may include a tangiblecomputer-readable storage medium on which is stored one or more sets ofinstructions (e.g., software) embodying any one or more of the methodsor functions described herein, including those methods illustratedabove. The instructions may also reside, completely or at leastpartially, within the main memory, the static memory, and/or within theprocessor during execution thereof by the computer system. The mainmemory and the processor also may constitute tangible computer-readablestorage media.

Dedicated hardware implementations including, but not limited to,application specific integrated circuits, programmable logic arrays andother hardware devices can likewise be constructed to implement themethods described herein. Application specific integrated circuits andprogrammable logic array can use downloadable instructions for executingstate machines and/or circuit configurations to implement embodiments ofthe subject disclosure. Applications that may include the apparatus andsystems of various embodiments broadly include a variety of electronicand computer systems. Some embodiments implement functions in two ormore specific interconnected hardware modules or devices with relatedcontrol and data signals communicated between and through the modules,or as portions of an application-specific integrated circuit. Thus, theexample system is applicable to software, firmware, and hardwareimplementations.

In accordance with various embodiments of the subject disclosure, theoperations or methods described herein are intended for operation assoftware programs or instructions running on or executed by a computerprocessor or other computing device, and which may include other formsof instructions manifested as a state machine implemented with logiccomponents in an application specific integrated circuit or fieldprogrammable gate array. Furthermore, software implementations (e.g.,software programs, instructions, etc.) including, but not limited to,distributed processing or component/object distributed processing,parallel processing, or virtual machine processing can also beconstructed to implement the methods described herein. Distributedprocessing environments can include multiple processors in a singlemachine, single processors in multiple machines, and/or multipleprocessors in multiple machines. It is further noted that a computingdevice such as a processor, a controller, a state machine or othersuitable device for executing instructions to perform operations ormethods may perform such operations directly or indirectly by way of oneor more intermediate devices directed by the computing device.

While the tangible computer-readable storage medium is shown in anexample embodiment to be a single medium, the term “tangiblecomputer-readable storage medium” should be taken to include a singlemedium or multiple media (e.g., a centralized or distributed database,and/or associated caches and servers) that store the one or more sets ofinstructions. The term “tangible computer-readable storage medium” shallalso be taken to include any non-transitory medium that is capable ofstoring or encoding a set of instructions for execution by the machineand that cause the machine to perform any one or more of the methods ofthe subject disclosure. The term “non-transitory” as in a non-transitorycomputer-readable storage includes without limitation memories, drives,devices and anything tangible but not a signal per se.

The term “tangible computer-readable storage medium” shall accordinglybe taken to include, but not be limited to: solid-state memories such asa memory card or other package that houses one or more read-only(non-volatile) memories, random access memories, or other re-writable(volatile) memories, a magneto-optical or optical medium such as a diskor tape, or other tangible media which can be used to store information.Accordingly, the disclosure is considered to include any one or more ofa tangible computer-readable storage medium, as listed herein andincluding art-recognized equivalents and successor media, in which thesoftware implementations herein are stored.

The illustrations of embodiments described herein are intended toprovide a general understanding of the structure of various embodiments,and they are not intended to serve as a complete description of all theelements and features of apparatus and systems that might make use ofthe structures described herein. Many other embodiments will be apparentto those of skill in the art upon reviewing the above description. Theexemplary embodiments can include combinations of features and/or stepsfrom multiple embodiments. Other embodiments may be utilized and derivedtherefrom, such that structural and logical substitutions and changesmay be made without departing from the scope of this disclosure. Figuresare also merely representational and may not be drawn to scale. Certainproportions thereof may be exaggerated, while others may be minimized.Accordingly, the specification and drawings are to be regarded in anillustrative rather than a restrictive sense.

Although specific embodiments have been illustrated and describedherein, it should be appreciated that any arrangement which achieves thesame or similar purpose may be substituted for the embodiments describedor shown by the subject disclosure. The subject disclosure is intendedto cover any and all adaptations or variations of various embodiments.Combinations of the above embodiments, and other embodiments notspecifically described herein, can be used in the subject disclosure.For instance, one or more features from one or more embodiments can becombined with one or more features of one or more other embodiments. Inone or more embodiments, features that are positively recited can alsobe negatively recited and excluded from the embodiment with or withoutreplacement by another structural and/or functional feature. The stepsor functions described with respect to the embodiments of the subjectdisclosure can be performed in any order. The steps or functionsdescribed with respect to the embodiments of the subject disclosure canbe performed alone or in combination with other steps or functions ofthe subject disclosure, as well as from other embodiments or from othersteps that have not been described in the subject disclosure. Further,more than or less than all of the features described with respect to anembodiment can also be utilized.

Less than all of the steps or functions described with respect to theexemplary processes or methods can also be performed in one or more ofthe exemplary embodiments. Further, the use of numerical terms todescribe a device, component, step or function, such as first, second,third, and so forth, is not intended to describe an order or functionunless expressly stated so. The use of the terms first, second, thirdand so forth, is generally to distinguish between devices, components,steps or functions unless expressly stated otherwise. Additionally, oneor more devices or components described with respect to the exemplaryembodiments can facilitate one or more functions, where the facilitating(e.g., facilitating access or facilitating establishing a connection)can include less than every step needed to perform the function or caninclude all of the steps needed to perform the function.

The invention claimed is:
 1. A three dimensional (3D) near eye displaydevice comprising: a display screen that displays a plurality of twodimensional (2D) images; a focusing element that collimates theplurality of 2D images; a spatial multiplexing unit (SMU) that remapsthe plurality of 2D images to different depths while forcing centers ofthe plurality of 2D images to align; and an eye piece.
 2. The 3D neareye display device of claim 1, wherein the SMU is selected from aliquid-crystal-on-silicon (LCOSD) special light modulator (SLM), avolume holography grating (VHG), and a distorted phase grating.
 3. The3D near eye display device of claim 1, further comprising a beamsplitter positioned between the display screen and the focusing element.4. The 3D near eye display device of claim 1, wherein the focusingelement is located between the display screen and the SMU.
 5. The 3Dnear eye display device of claim 1, further comprising a view combinerdevice that provides augmented reality viewing.
 6. The 3D near eyedisplay device of claim 1, wherein the focusing element comprises afirst lens.
 7. The 3D near eye display device of claim 6, furthercomprising a second lens, wherein the first lens is positioned betweenthe display screen and the SMU, and wherein the second lens ispositioned between the SMU and the eye piece.
 8. A method comprising:displaying, by a display screen of a wearable device, a plurality of twodimensional (2D) images; collimating, by a focusing element of thewearable device, the plurality of 2D images; modifying, by a spatialmultiplexing unit (SMU) of the wearable device, a phase of incidentlight by adding quadratic and linear phase terms to an incident wavefront of the 2D images resulting in multiplane images; and reimaging, byan eye piece of the wearable device, the multiplane images onto aviewer's retina.
 9. The method of claim 8, wherein the SMU is selectedfrom a liquid-crystal-on-silicon (LCOSD) special light modulator (SLM),a volume holography grating (VHG), and a distorted phase grating. 10.The method of claim 8, comprising reflecting, by a beam splitter of thewearable device, light of the plurality of 2D images towards the eyepiece.
 11. The method of claim 8, comprising overlaying, by a viewcombiner device of the wearable device, an image of a real world objectwith the multiplane images.
 12. The method of claim 8, comprisingapplying color filtering, by a color filter of the wearable device, tothe plurality of 2D images.
 13. The method of claim 8, comprisingapplying polarization, by a linear polarizer of the wearable device, tothe plurality of 2D images.
 14. A three dimensional (3D) near eyedisplay device comprising: a display screen comprising first and secondpanels that display two dimensional (2D) images; a beam splitter havingfirst and second surfaces that align with the first and second panels,respectively, wherein the beam splitter combines light emanating fromthe first and second panels of the display screen; an actuator thatlaterally slides the beam splitter in a direction along the first panelof the display screen to adjust a gap between the second panel and thesecond surface of the beam splitter resulting in an optical pathdifference; a focusing element that projects the 2D images to differentdepth planes; and an eye piece.
 15. The 3D near eye display device ofclaim 14, wherein the display screen comprises a single flexible displayscreen that is foldable into the first and second panels.
 16. The 3Dnear eye display device of claim 14, wherein the focusing elementcomprises a lens that is positioned between the beam splitter and theeye piece.
 17. The 3D near eye display device of claim 14, furthercomprising a view combiner device that provides augmented realityviewing.
 18. The 3D near eye display device of claim 17, wherein theview combiner device is selected from a waveguide, a beam splitter and aprism.
 19. The 3D near eye display device of claim 14, wherein the firstand second panels of the display screen are orthogonal to each other.20. The 3D near eye display device of claim 14, wherein the displayscreen is a flexible OLED.